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ABSTRACT 



The purpose of this research was to determine the problem 
solving behaviors of novice systems analysts during the design 
process. Using protocol analysis, this research found that novice 
analysts like their expert counterparts used an iterative problem 
solving process. However, unlike expert analysts, they exhibited 
a typical working behavior that tended to focus directly on the 
task at hand while overlooking larger but pertinent issues. 
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I . INTRODUCTION 

A. RESEARCH FOCUS 

This research focuses on the problem solving behavior of 
novice systems analysts during the systems design process. In 
this study the term "novice" is used to describe someone who 
possesses sufficient knowledge to function in a specific 
domain, but has limited practical experience in that domain. 
The findings will be compared to the problem solving behaviors 
of expert and novice financial analysts and expert systems 
analysts from the literature (Ramesh, 1989 ; Vitalari, 1981). 
The outcome of this research identifies differences in problem 
solving techniques of experts and novices. Possible uses for 
the differences will be discussed. 

B . SYSTEMS ANALYSTS 

What does a systems analyst do? A systems analyst may be 
compared to a language interpreter. The systems analyst 
functions as a mediator between the customer and the computer 
programmers and operators to facilitate effective 
communication. The systems analyst bridges the gap between 
the language, needs, and culture of the customer and those of 
the computer specialists (both hardware and software) . Thus 
the systems analyst's is required to be able to interface with 
the customer to understand separate business functions and 
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their intertwining relationships, and then to be able to 
represent this information in a format understandable to 
computer specialists for developing the system. The systems 
analyst is a designer, technician and artist all in one. 

1 . Growing Demand 

In today's competitive environment, information has 
become a strategic resource. Its prudent use, in many cases 
is essential to the very survival of organizations. This 
information requirement has exponentially increased the demand 
for systems capable of expeditiously representing data in 
formats compatible with an organizations' needs. As hardware 
becomes more capable and users become more aware of what 
computers can do for them, their desire for newer and more 
capable systems increases. This results in the already 
voluminous backlog of desired products. 

2. Difficulties Meeting Demand 

The demand for qualified and experienced systems 
analysts continues to grow. Currently this growing demand is 
hindered by the decreasing population from which new analysts 
can join the work force, and the extensive on-the-job 
experience currently required for a novice to progress to the 
level of expert systems analyst. The above problem is 
exacerbated by the fact that the size of the population under 
instruction required to maintain parity is directly 
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proportional to the duration of the instruction (Chi, Glasser 
and Farr, 19 88 ) . 

3 . Solution 

Many years of training and experience are required for 
a novice systems analyst to become an expert in his field. If 
properly addressed, the duration of this transition might be 
reduced. One suggested solution could be to identify the 
problem solving differences exhibited between expert and 
novice systems analysts during the design process. If these 
differences could then be incorporated into an educational and 
training process of systems analysts, the duration of the 
transition from novice to expert systems analyst could 
possibly be decreased. The decrease in the duration of the 
transition time would increase the supply of expert systems 
analysts, thereby lessening the demand. 

C. THESIS ORGANIZATION 

This thesis consists of five chapters. It is arranged to 
allow the reader to follow the development of the research 
from initial problem identification in Chapter I to proposed 
solutions in Chapter V. 

Chapter II provides a detailed description of the research 
design. Chapter III discusses the findings and contains a 
list of specific behaviors exhibited by novice systems 
analysts during this study. In Chapter IV the results of 
comparative studies between expert and novice systems analysts 
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and expert/novice financial analysts and novice systems 
analysts are provided. Chapter V, the conclusion suggests 
possible uses for the expert -novice differences identified 
during the comparative study. 
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II. RESEARCH DESIGN 



A. RESEARCH OBJECTIVE 

This study views and analyzes the problem solving behavior 
of novice systems analysts during a task performance. The 
results obtained will be compared with the problem solving 
behaviors of expert systems analysts and expert/novice 
financial analysts, as identified in other studies. These 
comparisons will identify if differences exist in the problem 
solving behavior of novice -novice or novice -expert analysts. 
The goal of this study is to identify these differences. 

The basis of this research is that the cognitive processes 
of individuals may be identified by analysis of task 
performance (Ericsson and Simon, 1984) . As the functionality 
of the cognitive processes is similar to that of an 
information processing system, a process model of the 
cognitive processes may be developed and analyzed (Vitalari, 
1981) . 

B . RESEARCH METHOD 

The methodology used in this study for capturing and 
analyzing the problem solving behavior of novice systems 
analysts during task performance is protocol analysis. A 
detailed description of this technique is provided in Appendix 
A. 
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C. SUBJECT SELECTION 

The first task when selecting the subjects for the study 
was to identify a specific population from which to choose the 
subjects. It was determined that the subject population 
should possess two specific attributes. The first was 
exposure to the theoretical procedures used by systems 
analysts during the systems design process. The second was 
minimal practical experience as systems analysts. 

The population identified as possessing these two required 
attributes was fifth quarter students in the Computer Systems 
Management (CSM) Curriculum at the Naval Postgraduate School, 
Monterey, CA. The first attribute requiring exposure to 
theoretical procedures used by systems analysts during the 
design process was satisfied by the subjects' classroom 
exposure in the CSM curriculum. While in this curriculum, the 
entire population received classroom instructions on systems 
design, database design and management, decision support and 
expert systems, programming, hardware and operating systems 
design, economic and financial considerations and managerial 
issues. In addition, as part of the instruction, the entire 
population participated in group and individual projects that 
required practical application of classroom theory. 

Volunteers were solicited. Eight students were selected 
to participate as subjects. They were all military officers. 
The group consisted of one Marine Corps officer and seven 
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Naval officers. Three of the subjects were Navy Lieutenant 
Commanders and the other five were Navy Lieutenants or their 
Marine Corps equivalent. 

Their undergraduate degrees were diverse in nature ranging 
from Computer Science to English. Only one subject had ever 
taken more than two computer related (specifically 
programming) courses prior to commencing their studies at the 
Naval Postgraduate School, Monterey, CA. 

Prior to attending the Naval Postgraduate School only one 
of the subjects had ever been involved with a systems 
development project in any capacity. That exposure consisted 
of functioning as a customer acceptance testing representative 
for the Navy on a systems development project. The subjects' 
minimal experience satisfied the second essential attribute. 

Though not a required attribute, it was interesting to 
note that all subjects, when asked, responded that they had 
never to their knowledge participated in a study that utilized 
Protocol Analysis. Each indicated that this was the first 
experience where they had been asked to think-aloud as they 
performed a task. 

D . TASK REVIEW 

The study was limited to the task of developing functional 
specifications during the Information Requirements 
Determination (IRD) phase of the systems development life 
cycle. The IRD phase may be considered to be the most 
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important segment of any systems development project. 
However, its importance is often overlooked or misunderstood 
as senior management considers the IRD phase to be an activity 
leading to an intangible product. Therefore sufficient time 
and resources are not allocated to the IRD. Budgetary 
constraints and the desire to have products marketed 
frequently lead to decisions to start implementation and work 
out the details or problems later. However, neglecting the 
IRD phase can prove to be disastrous. The literature is full 
of examples of systems development projects that were 
terminated or experienced significant delays and cost overruns 
when inadequate resources were placed into the IRD phase. The 
following are just a few examples: 

• Army Civilian Personnel System experienced a cost increase 
from the original 65 million projected to 96 million 
dollars (GAO, 1989) . 

• Defense Logistics Agency's Defense Logistic Service Center 
experienced a cost increase from the original 123 million 
projected to 177 million dollars (GAO, 1989) . 

• Air Force Contract Data Management System experienced a 
cost increase from the original 34 million projected to 74 
million dollars (GAO, 1989) . 

• Navy's Standard Automated Financial System experienced a 
cost increase from the original 33 million projected to 
479 million dollars before the project was terminated 
(GAO, 1989) . 

Therefore, the IRD phase should be considered to be 
crucial for successful systems development. The 

responsibility of convincing management to provide sufficient 



8 



time and resources for the IRD phase lies with the systems 
analyst. Furthermore, it is also the systems analyst's 
responsibility to ensure that developmental efforts during the 
IRD phase are competently and professionally conducted. 

Each subject was given the same task to perform. The task 
involved a case study (Appendix B) of a utility company's 
customer order processing system. This case study had been 
successfully used several times before in settings involving 
the use of protocol analysis for identifying problem solving 
behavior (Ramesh, 1989) . 

The task was to design a customer order processing system 
that utilized a centralized telephone answering service 
center, connected by an online computer to a large number of 
field stations that were responsible for providing services to 
the customers in their respective geographical areas of 
responsibility. The objective of the system was to provide 
quick response to customer requests for activities such as 
starting a service, stopping a service, switching a service, 
repairing appliances, setting up a location for services, 
removing services from a location and responding to power 
outages. The system should also provide real-time status of 
customer requests. The status would be used by utility 
representatives at the centralized telephone answering service 
center for answering customer inquiries about customer 
requests . 
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Included with the task was a description of the utility 
company's customer order processing system that was developed 
based on information obtained by a large systems consulting 
firm during an actual systems development proj ect (Ramesh, 
1989 ) . 

E. EXPERIMENTAL PROCEDURES 

The experimental sessions were scheduled to last 
approximately three hours. A separate session was scheduled 
for each subject. A facilitator was present for each entire 
session. The subjects could, if desired, take a short recess 
during the session. 

Each subject was provided with a copy of this information. 
The subjects were advised that the facilitator possessed more 
detailed information about the required system than that which 
was provided in their handout and could answer questions for 
clarification purposes. This option was only available prior 
to the initiation of the design process. 

1. Subject Briefing 

The first action during each session was to brief the 
subject about what he would be doing during the session and 
how the data he was going to provide would be used. The 
subject was then assured that the objective of the research 
was to understand rather than evaluate the problem solving 
behavior . 
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2 . 



Think-Aloud Familiarization 



The subject was then asked to participate in a 
structured exercise to provide him with the opportunity to 
become comfortable thinking -aloud while performing a task. 
First he was given instructions designed to have him talk- 
aloud while performing a task. Next, he was provided with a 
task (math and word problem) to perform. If additional 
practice was needed after the first task, subsequent tasks 
(word problems) were provided until the subject was 
comfortable with the technique. 

3 . Problem Introduction 

The subject was then given a handout with a 
description of the system to be designed. He was informed 
that the facilitator had additional information and could 
clarify issues upon request. Then he was asked to read the 
handout and hold any questions until finished with the initial 
reading. At that point any questions would be answered. Any 
clarification desired would be provided. Once the question 
and answer session was over, the subject was informed that he 
could retain the handout during the actual design process, but 
that the facilitator would no longer answer questions. He was 
also advised that the facilitator would be present only to 
remind him to think- aloud. It was suggested that if he had 
additional questions he should make whatever assumptions were 
necessary to proceed with his assigned task. 
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4. Designing the System 

Designing the system was the data gathering segment of 
the research. it was at this point that the tape recorder 
was turned on. 

The facilitator was present during the entire two-hour 
session. However, as stated before, his only function was to 
remind the subjects to think-aloud. 

5. Session Wrap-Up 

The subject was asked to fill out the personal 
information sheet located in Appendix C at the end of the 
session. The information requested pertained to the subject's 
educational and professional background. It also included a 
question about whether they had ever participated in a study 
that required them to think-aloud while performing a task. 

F. TRANSCRIPTION PROCESS 

All transcribing was performed by the facilitator. Each 
subject's entire recorded protocol was transcribed verbatim. 
Punctuation of the protocols was based on the facilitator's 
knowledge of word combinations and interpretation of the 
subjects' verbal inflections while speaking. 
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G. ENCODING PROCESS 

The results of the transcription were then used to develop 
a process model. Once the process model has been developed it 
can be used for determining the subjects problem solving 
behavior . 

To develop a process model from which to analyze the 
subjects problem solving behavior, the weakest hypothesis 
required is that the subjects' cognitive process during task 
performance is similar to the operation of an information 
processing system (Ericsson and Simon, 1984) . Data stored in 
the memory of a computer is in the form of electrical charges 
representing either ones or zeros. Without coding the ones 
and zeros are meaningless, but when encoded they represent 
data that is recognizable to a user. The human brain also 
represents stored data as electrical charges. For 
verbalization or output to take place the data in memory must 
first be encoded. By segmenting the verbalizations into their 
basic components and then coding them, one can use the coded 
data to develop a process model to be used for analysis. 

Protocols are frequently analyzed using episode 
representation. This involves splitting the protocols into 
topic segments representing a single task, and then analyzing 
these segments to identify specific kinds of activities or 
episodes (Ramesh, 1989). 

Episodic memory contains information about individual 
experiences that a person has had plus generalized 
episodes or types of events. Generalized episodes ... are 
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created by comparing different episodes and calculating 
similarities between them. . . According to Kolodner 
(1983) , "even if a novice and an expert have the same 
semantic knowledge ...the expert's experience would have 
allowed him to build up better episodic definitions... 
(Gilhooly, pp. 184-186,1986) 

In this research, episode representation was used with a 
minor variation to analyze the protocols. Rather than 
splitting the protocols into topic segments representing a 
single task and then identifying the specific episodes within 
the task segments, the episodes were identified and 
categorized directly from the protocols without any 
preprocessing . 

After the segmentation was completed each episode was 
analyzed independent of all other episodes and assigned a 
number representing the type of activity captured. This 
process was conducted twice. Results were compared and 
differences were reviewed before final assignment. 

The numbering system for the types of episodic activity is 
provided in Table 1. 

After all episodes were categorized, the numeric episode 
representations were separated from the protocols for 
comparison between the subjects. 
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Table 1: EPISODE TYPES KEY CODE 



EPISODE TYPE 


KEY CODE 


Identification of Relevant Data 


1 


Identification of Problems 


2 


Identification of Relationships between 
Problems 


3 


Generation of Hypothesis 


4 


Confirmation of Hypothesis 


5 


Assessment of Problem Criticality 


5 


Goal Setting 


7 


Reviewing Previous Actions 


1? 


Requesting Clarification from Customer 


9 


Making Assumptions 


10 


Disregarding Previous Actions 


11 


Reading from Case Write-Up 


12 


Intuitive Leap 


13 
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III. RESEARCH FINDINGS AND DISCUSSION 



A. INTRODUCTION 

During the encoding process, the following pattern of 
activities were observed. The subjects frequently performed 
independent activities, within the same episodic 
classification, for an extended period. The numeric 
representations of these episodes were first examined for 
consistency in coding within the activity period. This was 
done primarily for verification of the coding validity. The 
results of the evaluation of the coding scheme within the 
different activity periods were found by the research 
facilitator to be consistent throughout the entire process. 

The next activity was to examine episodic activity in 
sequence from first activity to last activity. The results of 
this indicated that all subjects performed the same types of 
activities in essentially the same order throughout the design 
process. A global evaluation of the numeric representation 
confirmed this conclusion, and indicated that the problem 
solving activity was an iterative process. The pattern of 
episodic activity indicated that the subjects had divided the 
design process into different levels and worked at each level 
in the same manner. 



16 



Based on the data encoded the remaining parts of this 
chapter focuses on the development of the process model and 
the specific behaviors of the subjects during task 
performance . 

B. PROCESS MODEL DEVELOPMENT 

The design process used by novice systems analysts 
consists of four phases. These four phases are the goal 
formulation phase, identification of relevant data phase, 
hypothesis generation phase and hypothesis confirmation phase. 
A graphic model of this process is provided in Figure 1. 

1. Goal Formulation 

The initial activity engaged in by each of the 
subjects was goal formulation. These goals were broad 
statements of intended actions. The goals were mostly 
concerned with the procedural aspects of developing a data 
flow diagram instead of identification of problems and 
developing solutions. 

While goal formulation was the initial activity of 
each subject, it was not restricted to the initial period of 
activity. Development of the initial goals into sub-goals 
occurred throughout the design process. The sub-goals were 
also primarily concerned with procedural aspects or the 
technicalities of the data flow diagram development process. 
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Figure Is Novice Systems Analysts' Process Model 



However, at the sub-goal level there were some goals that were 
concerned with the performance of activities within an element 
of the data flow diagram such as a specific process like 
"start a service". This dynamic process of formulating goals 
and sub-goals was probably the result of the iterative 
processes associated with the development of a data flow 
diagram. 

It was stated earlier that the goals of the subjects 
were very broad. This vagueness coupled with the magnitude of 
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the task and the numerous details that had to be accounted for 
may have contributed to the difficulty that all subjects 
experienced in keeping track of events not being directly 
acted on at the time. 

2. Identification of Relevant Data 

In this phase, the subjects extracted and examined in 
extensive detail all the information contained in the customer 
interview write-up. The relevant information was categorized 
as entities (internal or external) , data, or processes in the 
system. The subjects then set out to establish relationships 
between the data, the entities and the system in a format 
which would accommodate the desired functionality. 

It was during this phase that problems were 
encountered. These problems were associated with how to 
represent data or how a specific event would affect a larger 
whole. As previously stated they were very specific problems 
of limited scope. These problems were either resolved when 
discovered or set aside for future use. 

The subjects' interest in data was primarily 
quantitative. The search for data was a broad-based search. 
The attitude was look at everything, because all data were 
perceived as important. So much effort was expended looking 
at all of the data that there was little energy left for 
looking at any data in detail. Data were for the most part 
taken at face value. For example. There was an entity called 
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"customer". However, the customer could be a new customer or 
an existing customer. The utility's accounting and 
engineering departments could also be customers. None of the 
subjects was able to designed their system to include all the 
customer types through all the processes. Usually, only two 
of the four customer types were considered throughout the 
entire design process. However, each of the four customer 
types did require unique processing in at least one of the 
requests. This meant that duplication of efforts was not a 
valid reason for overlooking the different types of customers. 

3 . Hypothesis Formulation 

During this phase the subjects developed a single 
hypothesis for the specific process upon which they were 
working at the time. The hypothesis developed were directed 
towards getting a process requested by the customer through 
the various levels of the data flow diagram. Hypothesis were 
generated for processes at every level required to fully 
illustrate the processes satisfactory from start to 
completion . 

The subjects did not try to find a best way to build 
a system. Their action was to find any way to build a system. 
They would generate a hypothesis, and if that hypothesis 
passed testing, it was accepted. 

Another problem was that the subjects would generate 
a hypothesis, and then apparently forget about it. This could 
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be attributed to undefined goals or it may have been 
indicative of under developed indexing abilities in the 
systems analyst's domain. 

4. Hypothesis Confirmation 

The subjects design rational concerning hypothesis 
formulation and confirmation was very direct and very narrow 
in scope. The first action was to formulate a hypothesis. 
The subjects then determined if the hypothesis could be tested 
with data already extracted from the handout. If the 
hypothesis could not be tested with the available data, the 
subjects would search for more data. If the hypothesis could 
be tested with the available data, the subjects would test the 
hypothesis. If the hypothesis passed testing, the subjects 
confirmed the hypothesis. If the hypothesis failed testing, 
the subjects discarded the hypothesis and started the process 
over . 

The testing consisted of actually running the process 
or data through the entire developed portion of data flow 
diagram and verifying that all the necessary events occurred 
(desk- top testing) . If the hypothesis passed the test, it was 
confirmed and the subject continued with the iterative design 
process. If the hypothesis did not pass the test it was 
discarded and the subject would either formulate a new 
hypothesis right then or return to the data gathering phase. 
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C. IDENTIFICATION OF THE NOVICE ANALYSTS' BEHAVIORS 

Constructing the process model is just the first step in 
the analysis of the problem solving behavior of the novice 
systems analyst during the design process. In many ways it 
can be considered a quantitative evaluation, providing a 
global perspective from which to view the problem solving 
behavior. But, more importantly it provides an organized 
framework from which further analysis and qualitative 
evaluation of specific behaviors can be conducted. 

The results of this qualitative evaluation provides 
meaningful insights into the problem solving behavior of the 
systems analyst during the design process. This information 
may be compared to similar information about novices in other 
fields to identify systems analyst's unique attributes 
(Ramesh, 1989) . However, the greatest benefit from the 
information obtained from this analysis is that may be 
compared to results of similar studies of expert systems 
analyst to identify differences in expert-novice problem 
solving behavior (Vitalari, 1981) . 

The following are findings from the qualitative analysis 
of the process model. Novices: 

• Set broad, poorly structured goals. 

• Relied on their ability to apply the data to the 
methodology. 

• Frequently referenced the handout to ensure all the data 
was accounted for. 
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• Used a iterative design methodology. 

• Allowed methodology to lead them through the design 
process in an unorganized fashion. 

• Retained the same methodology throughout the design 
process . 

• Recognized the importance of user participation in the 
design process. 

• Recognized their shortcomings and the need to consult 
others about their activities. 

• Did not use heuristics. 

• Did not have preconceived expectations. They asked very 
few questions during the question answer period. 

• Did not readily recognize when errors had been made. 

• Spent little time checking their work. 

• Did not consider intra- organization politics during the 
design process. 

• Were not concerned about training and implementation. 

• Accepted data at face value. 

• Did not restrict their problem space to a manageable size. 

• Exhibited a lack of confidence in their action. 

• Did not evaluate the quality of their hypotheses. 

• Were able to identify complex relationships between data. 

• Were unable to incorporate their initial impressions of 
firm into their design process. 

• Used desk- top testing to confirm hypotheses. 

• Discarded hypotheses when data did not support the desk- 
top testing. 

• Did not seek problems or clues. Novices solved problems 
as they were encounter or set them aside for later 
resolution. 
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In Chapter IV these findings are compared with finding 
from earlier studies to identify expert-novice and novice- 
novice differences. 
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IV. ARE NOVICE SYSTEMS ANALYSTS DIFFERENT FROM THEIR 
EXPERIENCED COUNTERPARTS? - A COMPARATIVE STUDY 



To understand expert -novice differences, it is useful to 
study the characteristics of expert behavior as established by 
prior work. These characteristics can then be compared to 
those of novices in our study to identify areas of potential 
differences . 



A. CHARACTERISTICS OF EXPERTS 

Expertise is most often exhibited by individuals in a 
single domain and most characteristics that experts possess 
are necessarily domain specific. However, the following 
characteristics have been noted in most experts regardless of 
their domain. 



• Experts are specialists and exhibit their expertise 
primarily in the domain of their specialty. Intelligent 
people may be that way because of the indexing of Long 
Term Memory in their domain specialty rather than the 
global quality of their thinking (Minsky and Papert, 
1974) . 

• Experts exhibit superior perceptual skills within their 
domain. As shown in a study of GO players, experts are 
able to see larger patterns than novices (Reitman, 1976) . 

• Experts are faster and more accurate than novices. That 
can be the shown through the development of automatic 
skills by practice such as in typing which frees up memory 
capacity for the other task essential to typing such as 
reading or taking dictation (Chi, Glasser and Farr, 1988) . 
It may also be shown in the large meaningful patterns that 
experts perceive within their domain, therefore, 
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eliminating many of the search routines that novices would 
have to perform. Cab drivers will recognize a shorter 
route while traveling to their destination, even though 
the route was not identified in advance in a laboratory 
setting (Chase, 1983) . 

• Experts have expanded Short Term Memory within their 

domain through the automaticity of many portions of their 
skills. Normal Short Term Memory is 10-18 digits, 

however, trained memory experts can remember up to 80 
digits in short-term serial recall. (Chase and Ericsson, 
1982) . 

• Experts see through the superficial attributes of a 
problem to its deeper meaning. Expert programmers will 
group programs by algorithms while novices will group them 
by their application (Weiser and Shertz, 1983). 

• Experts will spend a lot of time before starting to solve 
the problem, just thinking about it, in order to obtain a 
better understanding of how they should go about solving 
the problem . Novices just tend to plunge in immediately 
and work towards a solution haphazardly (Chi, Glasser and 
Farr, 1988) . 

• Experts will check their work more often than novices and 
seem to better recognize when they have made errors that 
need correction before continuing (Chi, Glasser, and Farr, 
1988) . 



Ramesh (1989) used episodic knowledge obtained from the 
transcriptions of the verbal protocol to develop process 
models for representing the problem solving behavior of expert 
and novice financial analysts. He found that expert financial 
analysts solved their problems in three distinct phases 
referred to as the situation assessment phase, hypothesis 
generation phase, and the diagnostic evaluation phase. Within 
these distinct phases specific activities were exhibited by 
the expert financial analysts. The situation assessment phase 
included goal formulation, examination of key areas, and 
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formulation of initial impressions. The hypothesis generation 
included select and analyze relevant data. Finally, the 
diagnostic evaluation phase consisted of identification of 
causes, criticality assessment and identification of other key 
areas . 

These findings will be compared and contrasted with the 
results of the analysis from this study to identify any 
expert-novice differences. 

Vitalari (1981) used operator representation technique, 
involving the identification and use of knowledge elements and 
operator elements to study systems analysts' problem solving 
behavior. He categorizes expert behavior into different types 
of search behavior, problem perspective and focus. 

The different types of search behavior are the search for 
triggers or clues, search for goals which limit the magnitude 
of the search area, search for different strategies from which 
to approach the problem in order to maintain flexibility, and 
the search for applicable hueristics used for reducing the 
search time (Vatalari, 1981) . 

Experts were determined to have a global problem 
perspective (Vatalari, 1981) . This was determined to be 
consistent with the concepts of hierarchical decomposition and 
modularity associated with structured analysis techniques 
(Gane and Sarson, 1979) . 

The focus was primarily concerned with the different 
orientations from which to evaluated and solve the problem. 
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There was the report orien.te.tion., job-orientation, political- 
orientation, involvement -orientation, management -orientation 
and facilitation focus (Vatalari, 1981) . 

Expert-novice differences will be identified by comparing 
Vitalari's (1981) findings concerning how expert systems 
analysts developed and executed their search behaviors, 
problem perspectives and basic orientations to approaching the 
problem to findings in this study of novice systems analysts' 
problem solving behavior. 

B. CHARACTERISTICS OF NOVICE SYSTEMS ANALYSTS AND DESIGN 

Ramesh (1989) found that novices, like experts, solved 
problems in three distinct phases. These phases were 
different from that which the experts used in their problem 
solving behavior. The phases were problem identification, 
hypotheses formulation and final diagnosis. Each phase 
included different activities. The problem identification 
phase included goal identification and indication of problems. 
The hypothesis formulation phase activities were group related 
findings and identify consistent findings. The final 
diagnosis phase activities consisted of identification of 
causal linkages and final diagnosis (Ramesh, 1989) . 

The problem solving behavior of novice financial analysts 
identified in Ramesh' s (1989) study was compared to problem 
solving behavior of novice system analyst identified in this 
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study to identify novice-novice differences and 
characteristics unique to novice system analysts. 

The novice-novice differences that were identified are 
presented in the following section. 

C. NOVICE -NOVICE DIFFERENCES 

Novice- novice differences refers to the different behavior 
exhibited by members from different domains while performing 
a domain specific task. In order to identify novice-novice 
differences, novices' protocols from different domains are 
analyzed using the same method of representation. This 
methodology enables the researcher to identify the differences 
in the problem solving behavior between the two domains and 
also allows for generation of hypothesis about behavior which 
may be unique to a specific domain or common to all domains. 

While analysis of the problem solving behavior of members 
from two different domains is too limited a field to confirm 
hypotheses about the uniqueness of domain specific problem 
solving behavior, the analysis of the problem solving behavior 
of members from two different domains does allow for the 
formulation of hypotheses that can be investigated in future 
studies . 

1. Systems Analysts -Financial Analysts Differences 

Financial analyst problem solving behavior consisted 
of the problem identification, hypothesis formation and final 
diagnosis phases (Ramesh, 1989) . This differed from the 
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problem solving behavior of novice systems analysts in which 
four distinct phases consisting of goal formulation, gathering 
of relevant data, hypothesis formulation and hypothesis 
confirmation were identified. 

Identifying symptoms or clues to problems to be solved 
was a primary consideration of financial analysts (Ramesh, 
19 89) . This activity was not important to the systems 
analysts. The systems analysts were interested in obtaining 
data to support the development of a design following a 
methodology. Problems that systems analysts encountered were 
a hinderance to task performance, and not the primary 
consideration of the systems analysts. They were something 
that needed solving so that designing the system could 
continue . 

Goal formulation and identification of problems are 
episodic events within the financial analysts' problem 
identification phase (Ramesh, 1989) . The financial analysts' 
goals were not well defined and used generic terms (Ramesh, 
19 89) . The novice systems analysts' goals were not any 
clearer or better defined goals, however, they were technical 
in content. Novice systems analysts' goals indicated that 
they were going to commence an activity that was specific and 
structured. While novice systems analysts did not 
specifically state what they were going to do within framework 
of that structured methodology, the fact that they were going 
to participate in a very structured activity implies that the 
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primary orientation of novice systems analysts is technical 
with secondary consideration given to managerial factors such 
as the business environment and the overall strategy of the 
organization. This is different from the novice financial 
analysts who are concerned primarily with managerial issues 
such as health of the firm, business environment and strategy. 
The novice financial analysts use ratios, graphs and 
statistical analysis as tools to aid managers in their job of 
achieving the company goals. 

The primary difference in the hypothesis formulation 
phase was that the financial analysts would formulate a 
hypothesis which had to be justified by significant findings 
(Ramesh, 1989) . If the findings did not lead to the 
formulation of a plausible hypothesis, then the data would be 
discarded. The novice systems analysts' hypotheses did not 
require significant findings in the Hypothesis Formulation 
phase to justify its validity. In the Hypothesis Formulation 
phase of the novice systems analysts nothing was discarded. 

The hypothesis confirmation phase of novice systems 
analysts differs from the final diagnosis phase of novice 
financial analysts in several ways. The primary activity 
within the hypothesis confirmation phase of novice systems 
analysts is the testing of the formulated hypothesis. It is 
during this phase that a hypothesis will be discarded if data 
do not support it. There is no similar activity within the 
Final Diagnosis phase exhibited by novice financial analysts. 



31 



Also systems analysts do not attempt to integrate the 
findings. The systems analysts' task is an iterative process 
within the framework of the formal structured methodology. 
The fully developed structure is the final product. 

2. Systems Analysts Specific Events 

Systems Analysts were unique in at least two aspects. 
During goal formulation the novice systems analysts committed 
to a formal structured methodology for performing the assigned 
task. Second, after the systems analysts formulated goals 
they would gather data and formulate one hypothesis that fit 
the data to the structure and then test the hypothesis. A the 
successful testing resulted in hypothesis confirmation. This 
format in which the novice system analysts would formulate a 
hypothesis, test the hypothesis and then confirm or discard 
the hypothesis based on the ability of the data to support the 
hypothesis was unique to systems analysts. 

D. EXPERT-NOVICE DIFFERENCES 

The next sections describe the expert-novice differences 
obtained from comparisons of novice systems analysts' problem 
solving behaviors with expert financial/systems analysts' 
problem solving behaviors . 

1. Episode Representation Differences 

The comparison of novice systems analysts and novice 
financial analysts discussed earlier indicated that there was 
sufficient similarity for comparison between the two groups. 
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Ramesh's (1989) work with expert financial analysts using 
episode representation to model and analyze their problem 
solving protocols provides an excellent opportunity for such 
a comparison. In this section we will compare behavior of 
expert financial analysts and novice systems analysts. The 
major differences between the two categories are as follows: 



• Experts stated their goal clearly and referred to them 
frequently. Novice goals were broadly stated, technical 
in wording and not frequently referred to after initially 
stated. 

• Experts started their analysis with an initial impression 
of the firm which helped them form expectations and 
develop list of potential problem areas. Novices form 
initial impressions of the firm and then disregarded them 
once the design process started. 



Experts Spend a Great Deal of Time Analyzing a Problem 
Qualitatively. Protocols show that, at the beginning of 
a problem-solving episode, experts typically try to 
"understand" a problem, whereas novices plunge immediately 
into attempting to apply equations and to solve for an 
unknown. What do the experts do when they qualitatively 
analyze a problem? Basically they build a mental 
representation from which they can infer relations that 
can define the situation, and they add constraints to the 
problem.... (Chi, Glasser and Farr, pp. xvii-xx, 1989) 



• Experts searched for clues to support or negate their 
hypothesis during the hypothesis generation phase. 
Novices accepted their hypotheses at face value. Novice 
system analysts did not evaluated their hypotheses until 
the Test Hypothesis phase. 

• Experts looked for "good" and "bad" signals in the data 

they analyzed to be used for correcting initial hypothesis 
in case there were discrepancies. When a novice 
formulated a hypothesis it was accepted as is or 
discarded. The quality of the hypothesis was not a 

consideration. 
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• Experts were able to identify and maintain in their minds 

complex relationships between various segments while 
analyzing data. Novices could identify complex 

relationships within segments and across various segment, 
but could not maintain the relationships while analyzing 
the data. 

• Experts had preconceived ideas of "key areas" that 
required examination. This was probably the result of 
past experience in solving similar problems. Novices did 
not assign special meanings to particular areas, but 
rather looked at all areas in the order that was presented 
in the case write-up. Novices have very limited practical 
experience, therefore, they are not able to generated 
preconceived ideas of what to expect or look for, but must 
look at every thing in total . 

• Experts ranked their hypothesis in order of importance. 
Novices did not rank their hypothesis. There were 
indications that the novices recognized that certain 
hypothesis were more important than others, but these 
hypothesis were not given any special consideration. 

• Experts were interested in the firm's goals and 
strategies, both internal and external to the 
organization. Novices' interests in the firm were related 
to how the customer service system would interact with 
different functional areas internal to the organization. 

• Experts were interested the managerial style of the firm 
and the business environment in which the firm operated. 
Novices were only interested in the system which they were 
designing. 



2. Knowledge Operator Differences 

Nicholas P. Vitalari (1981) used knowledge operator 
representation to analyze the problem solving protocols of 
expert and novice system analysis. The following behavioral 
differences were the results of a comparison between the 
findings of this study and Vitalari' s work. Experts: 

• Set specific well structured goals. Novice set broad 
poorly structured goals. 
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• Developed multiple strategies for achieving the goals. 
Novices did not develop a strategy. They allowed the 
methodology to lead them through the design process in an 
unorganized fashion. 

• Would modify or discard a strategy when evidence indicated 
that the goals were not being achieved. Novices stayed 
with the same methodology throughout the process. 

• Applied heuristics during the problem solving process. 
There was no indication that the novices used heuristics 
at any time during the process. 

• Were aware and showed consideration for the political 
attributes of the firm such as territorial consideration 
and orders of priority on who initiated and responded to 
various actions. Novices recognized that there were 
different areas of authority, but treated each as an equal 
partner . 

• Were concerned about the organizational behavior such as 
the "resistance to change" phoneme. Novices were only 
interest in designing a system. They did not consider how 
this would affect the organization. 

• Were concerned about the skills of the individuals within 
the organization. Novices did not show any interest in the 
skills of the people within the organization. There was 
no concern for staffing requirements for operating and 
maintaining the system after implementation. 

• Were interested in how the secondary data could be 
formatted and used by upper management in addition to the 
uses at the operational level. Novices were only 
concerned with getting a workable system to the 
operational level. 

• Showed concern for more than the end product. They were 
interested in things like peoples names showing a personal 
orientation. Novices were only interested in the product. 

• Searched for clues about the problem for use in 
formulating goals and strategies. The novices took all 
the data at face value. 

• Had expectations and their search is characterized by what 
is missing instead of what is apparent. Novices search 
for data is a rehashing of what is apparent. 

• Were interested in the organization structure for use in 
developing maps and strategies to use during the problem 
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solving. Novices relied on the structure of the 
methodology and how raw data fit into the structure. 

• Used goals to manage the size of the problem space. 
Novices did not set good goals and it was apparent from 
the data loss during the design process that the problem 
space was larger than the novice could effectively manage. 

• Exhibited behavior which is meant to reduce uncertainty. 

Novices exhibited a lack of confidence in their ability to 
design the system. Their action increased the 

uncertainty. 
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V. CONCLUSION 



This study of the problem solving behavior of novice 
system analyst had three objectives: 

• To analyze the problem solving behavior of the novice 
system analysts during the design process. 

• To compare the results of this study to those of previous 
studies involving experts and identify the differences. 

• To suggest possible uses for the differences. 

Protocol Analysis provided a methodology for the analysis 
of the problem solving behavior of novice system analysts. 
Similar studies have been conducted using experts. Ramesh's 
(1989) studies of expert financial analysts using episode 
representation and Vitalari's (1981) work with expert systems 
analysts using knowledge operator representation were two of 
these studies. 

The findings of this study suggest that there are 
significant differences between expert and novices, in the way 
they perform systems analysis and design. Suggested ways to 
use these differences are provided in the following 
paragraphs . 

The first and most obvious use for the findings is in the 
traditional instructor- student relationship. By calling the 
experts behavior the goal and the novices behavior the current 



37 



situation, educational programs can be designed and 
implemented to practice those specific things that experts do 
and that novices do not do. Specifically efforts are directed 
towards known activities in which experts participate. 

The second and more remote use of the findings is in the 
development of intelligent tutoring systems. This method of 
education does not use a human instructor, but rather the 
student and a machine interact in a learning environment. 

Currently, the bottleneck in the development of effective 
expert systems is in the area of adequately modeling the 
behavior of experts for the systems. The availability of the 
behavioral models and expert-novice differences could assist 
with the development of expert or intelligent tutoring systems 
for systems analysts. 

Third, CASE technology currently provides users with tools 
that automatically generate code, create and maintain a data 
dictionary, write and update all required documentation and 
performs other tasks currently associated with systems 
development just by using various data flow and entity 
diagrams as input. The output is consistent and technically 
correct, however, the output is only as good as the diagrams 
that are used as input to the CASE tool. If the quality of 
the analysis and diagram development used for input to the 
CASE tool are poor, then the resultant output is a consistent 
and technically correct poor system. 
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A potential use for the results of this research is to 
build an expert system as a front-end processor to a CASE 
tool. The front-end processor would be used to evaluate the 
level of expertise of the analyst using the tool. By 
understanding the differences between the skills of the 
analyst providing the input and an expert analyst's skills, 
the front-end processor can improve the quality of input 
thereby ensuring the code and other generators operate using 
the best possible input. 
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Appendix A: AN OVERVIEW OF PROTOCOL ANALYSIS 



A. PROTOCOL ANALYSIS DEFINED 

A generic definition of Protocol Analysis is given below. 

Protocol analysis is a general term for the collection and 
analysis of verbal reports made by subjects while they 
perform a task. (Vitalari, p. 81, 1981) 

Ericsson and Simon (1984) broaden the scope of this 
definition by allowing for two types of verbalization. The 
first is a verbal report where subjects talk aloud while 
performing a task, referred to as concurrent verbalization. 
The second type of verbalization is a report made by subjects 
subsequent to the performance of a task, referred to as 
retrospective verbalization. 

B. OVERVIEW OF THE COGNITIVE PROCESS 

A primary assumption behind Protocol Analysis is that an 
individual's problem solving behavior can be determined from 
analysis of the subject's verbalizations. These 
verbalizations consist of interactions between information 
contained in Short Term Memory (STM) and Long Term Memory 
(LTM) . 

1. Short Term Memory 

STM is a recording of events and stimuli which an 
individual has experienced recently. STM capacity is minimal, 
though there is some variance between individuals. It is 
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instantly available, not requiring retrieval delay. STM 
duration can vary from that of a fleeting nature to moments. 
However, the longer the information that an individual is 
consciously aware of, or "heeded" remains in STM the greater 
the probability that it will become contaminated by the 
information previously stored in LTM, thereby losing its 
original identity. This does not preclude the possibility 
that the "heeded" data may have already been stored in LTM 
prior to contamination. The probability of data storage in 
LTM is dependent upon the duration that it is "heeded" in STM. 

It is even more obvious that information can be retrieved 
from LTM only if has been stored there previously, and 
retained. The hypothesis about storage (fixation) that 
seems to us most defensible in the light of the empirical 
evidence is that if and only if information is heeded in 
STM for a sufficient interval of time will it be stored 
(and indexed) in LTM. (Ericsson, Simon and Herbert, p.81, 
1984) 

Finally, STM is the gateway through which all interactions - 
passive and active with LTM are routed. 

2 . Long Term Memory 

LTM is an accumulation of all the events or stimuli 
that an individual has experienced and "heeded" during their 
life. Access to LTM compared to STM is slow yet, its capacity 
is virtually unlimited. 

Demonstrated ability to retrieve obscure facts from 
the voluminous amount of information that an individual has 
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stored in LTM over a lifetime of experiences seems to imply 
that LTM is indexed in some fashion. This indexing or 
categorization in many instances is highly organized and can 
be rapidly retrieved as demonstrated in studies of the "tip of 
the tongue" (James, 1890) and "felling-of -knowing" (Woodworth, 
1938) phenomena. 

A simplified view of accessing or retrieving 
information from this indexed or categorized LTM is to 
visualize a set of symbols formulated in STM which cause 
electrical impulses to activate locations in LTM. As a result 
data in the activated locations are copied into STM. This 
process, depending on a variety of factors, can be almost 
instantaneous, approaching the time required to access STM or 
it can proceed over an extended period. The degree of 
accuracy of the data retrieved from LTM can vary. This 
variance occurs because the indexing or categorization is 
dependent upon the relationships between data stored in LTM. 
The longer the data are stored in LTM, the greater the 
probability that the boundaries between the relationships that 
characterize different but similar data will merge. This 
merging of boundaries results in data stored in LTM acquiring 
characteristics not present when initially "heeded" . 
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C. PROTOCOL ANALYSIS AS A RESEARCH METHOD 

Protocol Analysis is a valid methodology for identifying 
the cognitive processes of individuals during task performance 
(Ericsson and Simon, 1984) . It provides the researchers with 
several ways to analyze problem solving behavior. 

1. Types of Protocol Analysis 

Protocol analysis is typically conducted using two 
different techniques. The techniques can be used separately, 
or in conjunction with, each other. The first method, 
retrospective analysis, requires the subject to perform a 
task, and after the task has been completed the subject 
verbalizes what he is thinking about as he performs the task. 
The second method, concurrent analysis, requires the subject 
to verbalize as he is performing the task. When the two 
methods are used in conjunction, the subject is asked to 
verbalize while performing the task, and then after task 
completion the subject is asked to reflect on his thoughts 
during task performance. 

The effectiveness of these two analysis techniques are 
briefly discussed below. 

a. Retrospective Analysis 

Retrospective analysis is very susceptible to 
inaccuracies. The primary reason is that retrospective 
analysis requires the subject to verbalize about an event that 
has occurred sometime in the past. If the event that the 
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subject is responding to occurred in the recent past, portions 
or even all of the "heeded" data may still reside in STM. 
However, as stated earlier, the possibility for contamination 
of data in STM increases as the duration of "heeded" data in 
STM increases. If the data to be verbalized are no longer in 
STM, then the subject has to initiate the retrieval of the 
desired information from LTM to STM for encoding and 
verbalization. In summary, information retrieved from LTM may 
be inaccurate for several reasons : 

• The information that has been stored in LTM is not always 
initially retrievable in total. 

• Over time, the boundaries in LTM weaken resulting in the 
merging of data of a similar nature. 

• The subject may misunderstand what is requested and 
provide inaccurate data, even though it may be relevant. 

b. Concurrent Analysis 

Concurrent analysis is not as susceptible to the 
inaccuracies associated with retrospective analysis. Since, 
the subject verbalizes his thought process as he performs the 
task, there is little chance that the information is 
contaminated . 

2. Issues in Using Protocol Analysis as a Research Method 

The purpose of this thesis is to use protocol analysis 
to analyze the problem solving behaviors of novice systems 
analysts during task performance. However, there are several 
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concerns about the validity of protocol analysis as a research 
method that need to be addressed first. These concerns are: 

• Doubts have been expressed about using verbalizations as 
data (Ericsson and Simon, 1984) . 

• Concerns have been raised about the spontaneity of the 
subjects verbalization (Ericsson and Simon, 1984). 

• It has been suggested that the verbalizations are "soft" 
data e.g. they are subjective and not measurable (Ericsson 
and Simon, 1984) . 

• Theoretical presuppositions during the encoding process 
have to be considered (Ericsson and Simon, 1984) . 

• Can the subject be trusted to not to be deliberately 
misleading with their verbalizations (Ericsson and Simon, 
1984) ? 



The following sections address these concerns, 
a. Response to Doubts 

The doubts expressed by many psychologists about 
the suitability of subjects verbalizations as scientific data 
have to do with introspection. Psychologists argue that 
retrospective analysis is a variant of the discredited process 
of introspection (Ericsson and Simon, 1984) . However, there 
is a significant difference between protocol analysis and 
introspection. Protocol analysis requires the subjects to 
verbalize their thoughts as they occur. This is possible 
because of the invention and availability of tape and video 
recorders. On the other hand, introspection involves a 
trained domain specialist, usually a psychologist, reflecting 
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on his own problem solving activities and writing or dictating 
to a stenographer an analysis of these activities. This 
process has all the previously mentioned problems associated 
with retrospective analysis, in addition to any preconceived 
biases of the person doing the introspection. 

Jb. Influencing Verbalization 

To obtain data suitable for the type of protocol 
analysis selected for use, a methodology must be developed to 
influence the subject's verbalization. Through careful 

wording of the instructions, different kinds of verbalizations 
can be obtained from a subject. Examples of the instructions 
are : 

• "I would like you to say out loud what ever thoughts come 
into your mind, as if you were in a room alone talking to 
yourself . " 

• "I would like you to talk aloud as you are performing the 
task . " 

• "I would like you to verbalize what you are doing as you 
are doing it and why you are doing it" 

The first two constructions are structured to obtain responses 
suitable for concurrent analysis. The third construction 
seeks to obtain a hybrid response for use in concurrent and 
retrospective analysis. 

c. "Soft" Data Versus "Hard" Data 

Data collected by verbalization are considered 
"soft" data primarily by people who misunderstand how the 
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verbalizations are going to be interpreted. "Soft" data are 
obtained when data interpretation occurs simultaneously with 
the verbalizations. By using this method the interpretation 
is subjected to the theoretical presuppositions of the 
interpreter (Ericsson and Simon, 1984) . 

"Hard" data can be collected and analyzed 
objectively. An example of "hard data" would be the recording 
of the time between eyelid movements during task performance. 
When using protocol analysis the protocols are recorded and 
transcribed verbatim. This data are just as "hard" as the 
data about eyelid movement (Ericsson and Simon, 1984) . 

d. Theoretical Presuppositions During Encoding 

With protocol analysis, a subject speaks out loud. 
The verbalization is recorded and then transcribed verbatim. 
At this stage there are no interpretations and no inferences, 
with the exception of some processing as verbal inflections 
and emphasis are represented by punctuation. 

To interpret "hard" data obtained from subjects, 
verbalizations need to be encoded. The coding process, just 
by its very nature, is subject to some theoretical 
presuppositions. The objective when analyzing the subjects 
verbalizations is to minimize the theoretical presuppositions 
to facilitate providing an objective as possible analysis of 
the subjects' behavior. This is done, first by minimizing 
inferences and expectations used by the person developing the 
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coding scheme. Further, it is accomplished by ensuring the 
person or persons doing the actual coding of the protocols 
were not involved with the development of the coding scheme. 
Finally, the protocols should be coded by at least two people 
working independently. The results of the codings are then 
compared and any differences are resolved, 
e. Trusting the Subject 

A basic but absolutely essential concern is how to 
ensure that the subjects are not purposely misleading 
researchers during the process (Ericsson and Simon, 1984) . If 
the subject was asked in an introspective report about his 
mental state or thought processes, trust is important and some 
validation is crucial (Ericsson and Simon, 1984) . With 
protocol analysis, researchers can eliminate the need to trust 
the subjects and collect the data required for determining 
problem solving behavior during task performance during the 
same process. 



However, the issue of the reliability of self- 
reports can (and, we think, should) be avoided 
entirely. The report "x" need not be used to infer 
that X is true, but only that the subject was able to 
say "X"_(i.e., had the information that enabled him to 
say "X".) by following this path, we can even show 
that there is an inverse relation between how much 
subjects need to be trusted and how much information 
they verbalize. For the more information conveyed in 
their responses, the more difficult it becomes to 
construct a model that will produce precisely those 
responses adventitiously-hence the more confidence we 
can place in a model that does predict them. 
(Ericsson, Simon and Herbert, p.7, 1984) 
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D. SUMMARY 



Verbalization is an activity of the cognitive process that 
is recordable for analysis. Protocol analysis provides the 
means to capture and analyze verbalizations during task 
performance in order to identify problem solving behavior. It 
is a methodology in which minimal theoretical presupposition 
is required for use, thereby, increasing the objectivity of 
the analysis of the data. When using protocol analysis the 
issue of trusting the subjects is not a concern. These 
factors make protocol analysis a valuable tool that should 
continue to be used and refined. 



49 



Appendix B: CUSTOMER SERVICE SYSTEM 



Southeast States Power, a gas and electric utility company, is 
automating a major part of its customer service system. The 
proposed system should utilize a centralized telephone 
answering service center, connected by an online computer to 
a large number of field stations. The central system will be 
installed with the objectives of providing quick processing of 
requests and providing accurate information to customers about 
the status of their requests. In the centralized set up, 
calls from customers will be routed to any clerk available at 
the service center. Further, requests for service may 
originate from within the utility itself. The accounting 
department may request cancellation or restoration of services 
based on the payment profile of customers. The engineering 
department may request setting up or removal of services from 
a location. 

The proposed system will support the following types of 
request : 

• starting a service 

• stopping a service 

• switching a service from one location to another 

• setting up a location for service 

• removing service from a location 
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• restoring service to a customer with local outage 

• restoring service to customers after system-wide outage 

• maintaining and repairing appliances 

The clerks generate service orders based on the request 
for service. Records maintained by the system contain 
information about the customers, location, equipment and 
appliances at the location, a and status of system-wide 
service. The clerks retrieve relevant information from 

internal records while generating service orders. The clerks 
submit the service orders to the service center supervisors 
for their authorization. the supervisors forward authorized 
service orders to field station supervisors for follow up. 

While processing a request from a customer for starting a 
service, the clerk obtains authorization from the credit 
department. Further, the clerk ascertains the need for 
requesting a deposit from the customer. If the customer is 
requesting that the service be stopped, the clerk should 
terminate any contracts for maintenance of appliances that 
have been entered into with the customer. The requests for 
switching service from one location to another will involve 
starting a service in the new location and stopping the 
service at the old location. While processing the request for 
setting up and removing service from a location, the clerk 
obtains details about the location and equipment used in the 
location so that internal records can be updated. When a 
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customer reports an outage, the status of the system- wide 
service should be checked before orders are placed for follow 
up. When requests for repairs of appliances are received, the 
clerk checks appliance maintenance contracts to ascertain 
charges (if any) for customer billing. 

Field station supervisors set priorities on service orders 
and then group them based on the location and nature of 
service to be provided. Field technicians with necessary 
skills (such as appliance repair, equipment set up etc.) are 
dispatched to provide service. After providing the service, 
field technicians report the status of the order (such as the 
nature of the service provided and completion status) to the 
field supervisors. Field supervisors reconcile this 
information with the service orders. Appropriate charges (if 
any) are computed and communicated to the accounting 
department for customer billing. 

The clerks provide information about the status of the 
service requests upon inquires from customers. They also 
notify service center supervisors when complaints about 
delayed or inadequate services are received. 



CUSTOMER SERVICE SYSTEM PROCESS DESCRIPTION 
CALCULATION OP DEPOSITS 

Compute the deposit to be paid by the customer based on 
the type of customer (residential or industrial) , the past 
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consumption pattern (if available, is maintained for a 12 
month period) or expected consumption as follows: 

Industrial Customer 

The deposit required is based on the maximum monthly- 
consumption from the consumption history. The actual value 
is : 

max_consumption * a constant (say 1.5) * approximate rate per 
unit (say $0.1 per unit) . 

When the history is not available, the maximum monthly 
consumption is computed as proportional to the installed 
capacity of the industrial unit. The actual value is: 
Installed capacity * a constant (say 2.0) * approximate rate 
per unit (say $0.1 per unit) . 

Residential Customers 

the deposit required is based on the average monthly 
consumption from the consumption history. The actual value 
is : 

average consumption * a constant (say 2.0) * approximated rate 
per unit (say $0.08 per unit) 

When the history is not available, the average monthly 
consumption is based on the type and size of the 
house/apartment to which the service is provided. Expected 
consumption is maintained as a table of the following form: 
Type of House Size (room) Consumption 
Town House 4 300 units 

Condo 2 100 units 
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The actual value is: 

Average consumption * a constant (say, 2.0) * approximate rate 
per unit (say, $0.8 per unit) 

(The following information was not given to the subject. It 
was used for clarification during the question answer 
session. ) 

CUSTOMER SERVICE SYSTEM CLARIFICATION FROM CLIENT INTERVIEW 



The proposed system will support the following types of 
request : 



• starting a service: starting a gas/electric service as a 
customer's location. 

• stopping a service: stopping a gas/electric service at a 
customer's location when the customer moves out of the 
area serviced by the utility company. 

• switching service from one location to another: when a 

customer moves from on location (old location) to another 
location (new location) that is also service by the 
utility company, service is discontinued in the old 
location and started in the new location. 

• setting up a location for service: setting up a location 
( such as installing appliances, meters etc.) so that 
gas/electric service can be provided to customers who will 
move into the locations. The requests for setting up a 
location originate from the engineering department. 

• removing service from a location: removing equipment such 
as appliances or meters from a location so that the 
location will not be serviced by the utility. The 
requests for removing service from a location originate 
from the engineering department. 

• restoring service to a customer with local outage: local 
outages are outages that affect the customer only. 
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• restoring service to customers after system- wide outage: 
system-wide outages are outages that affect a large number 
of customers. 

• maintaining and repairing appliances: stoves, furnaces, 

ovens etc. are examples of appliances. 

• Each field station provides service to part of the total 
geographical area serviced by the utility. 

• Field supervisors obtain information about the status of 
the orders from technicians and enter this information in 
the system. This status information is available to 
telephone answering center clerks for answering queries 
from customers. 



TELEPHONE SERVICE CENTER 



• The clerk receives requests from customers as well as the 
accounting and engineering departments of the utility. 
Activities leading to the generation of requests at the 
engineering and accounting departments is beyond the scope 
of the system. 

• The clerks processes the requests and sends them to 
supervisors for authorization and further action. 

• The clerks have access to credit information about the 
customers (that is created an maintained by the credit 
department) . This credit information specifies whether 
any deposit is required from the customer while processing 
the request. The amounts of deposit is calculated by the 
system using a pre- specif ied formula. The clerk informs 
the customers the amount of deposit and payment plans. 
The billing is handled by the accounting department. 

• When new customers whose credit information is not 
available request that a service be started, the clerk 
send a request for credit authorization to the credit 
department. Further, the clerk processes the request from 
the customer and forwards it to the supervisor for follow 
up . 

• A customer will have separate billing accounts for each of 
the locations in which he/she gets service. The clerk 
obtains the name and address of the customer or the 
account number to process requests. 
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• A customer may call to ascertain the status of a request 
made earlier. if the customers complain about delays in 
service or inadequate services, the clerk forward these 
complaints to the supervisors. 

• Status of system-wide service is available to the clerks. 
Customer complaints information is not used in determining 
whether a system-wide problem exists. 

• For each appliance installed at a customer's location, 
information on their type, installation date, service 
history, maintenance contract details etc. is maintained. 
The clerk checks the maintenance contract information to 
determine whether customer needs to be billed for 
appliance repairs. 

• Telephone answering center supervisors authorize the 
service order and forward them to field supervisors. 

• The credit department forwards credit authorization for 
new customers to the supervisors. Supervisors contact the 
customers when authorization information is received to 
inform them the status of their request as well as deposit 
requirements . 



FIELD STATION 



• Field station supervisors assign priorities to service 
orders according to a pre- determined scheme. 

• Supervisors assign one technician to process each order. 

• When the technician report on the status of the order, 
their supervisors reconcile the information with the 
service orders. further, based on the information, the 
system computes appropriate charges (if any) according to 
a pre -determined formula for customer billing. 

• The information on the status of the service orders is 
available to telephone answer center clerks for answering 
queries from customers. 
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Appendix C: PERSONAL DATA QUESTIONNAIRE 



1. What is your name? 

2 . What is your branch of service? 

3. What is your community? (Aviation, Surface- Line , 
Submarines, Supply, Intelligence, other)? 

4 . What is your curriculum and current quarter? 

5. What was you undergraduate major? 

6. Have you had and computer training prior to attending 
Naval Post Graduate School? If yes, was it design, 
programming, databases, other? How many quarters or 
semesters? 

7. Have you ever worked on Systems Development Project? If 
yes, in what capacity? 

8. Have you ever been a subject in a study utilizing protocol 
analysis? 
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